An Efficient Storm Identification from Big Rainfall Data Using MapReduce

This paper is part of my doctoral dissertation, “Analysis and Modeling Techniques for Geo-Spatial Datasets,” which focuses on how to summarize, model, and format spatiotemporal data for analysis and mining. The dissertation consists of four main components: (1) spatio-temporal knowledge representation, (2) identifying meaningful concepts from raw data, (3) converting raw data to conceptual data, and (4) analysis and mining of conceptual data. This paper, part of the third component, describes an efficient MapReduce algorithm for converting raw rainfall data into meaningful storm information, which can then be easily analyzed and mined. Our previous work proposed a method to identify relevant storm characteristics from raw rainfall data. The original storm identification system takes too long to produce the summarized storm characteristics, because: (1) the raw rainfall data, which is considered as big data, is stored in a traditional relational database based on CUAHSI (Consortium of Universities for the Advancement of Hydrologic Science, Inc.) ODM (Observations Data Model), which leads to substantial disk I/O; (2) the storm identification algorithm is based on recursion and regular depth-first-search (DFS), which leads to multiple retrievals for parts of the data. In this paper, we obtain a substantial improvement in performance by utilizing MapReduce. We also utilize the original raw rainfall data text files instead of using the data in the relational database. In our experiments, the performance of the new storm identification system is significantly improved compared to the previous one. With this new system, it will dramatically benefit hydrologists in helping them performing rainfallrelated analysis (both location-specific and storm-specific) such as flood prediction using our identified storms. Keywords-storm analysis; rainfall; big data; MapReduce; distributed computing; CUAHSI